| H0 is true | H0 is false | |
| Accept H0 | Type II error, miss | |
| Reject H0 | Type I error, false alarm |
Eva Freyhult
NBIS, SciLifeLab
2022-09-13
Statistical inference is to draw conclusions regarding properties of a population based on observations of a random sample from the population.
To perform a hypothesis test is to evaluate a hypothesis based on a random sample.
Typically, the hypotheses that are tested are assumptions about properties of a population, such as proportion, mean, mean difference, variance etc.
There are two hypotheses involved in a hypothesis test, the null hypothesis, \(H_0\), and the alternative hypothesis, \(H_1\).
\(H_0\), the null hypothesis is in general neutral, “no change”, “no difference between groups”, “no association”.
In general we want to show that \(H_0\) is false.
\(H_1\), The alternative hypothesis expresses what the researcher is interested in “the treatment has an effect”, “there is a difference between groups”, “there is an association”.
The alternative hypothesis can also be directional “the treatment has a positive effect”.
| H0 is true | H0 is false | |
| Accept H0 | Type II error, miss | |
| Reject H0 | Type I error, false alarm |
Type I error is a false positive, a false alarm that occurs when \(H_0\) is rejected when it is actually true. Examples: “The test says that you are covid-19 positive, when you actually are not”, “The test says that the drug has a positive effect on patient symptoms, but it actually has not”.
Type II error is a false negative, a miss that occurs when \(H_0\) is accepted, when it is actually false. Examples: The test says that you are covid-19 negative, when you actually have covid-19”, “The test says that the drug has no effect on patient symptoms, when it actually has”.
| H0 is true | H0 is false | |
| Accept H0 | Type II error, miss | |
| Reject H0 | Type I error, false alarm |
The significance level, \(\alpha\) = P(false alarm) = P(Reject \(H_0\)| \(H_0\) is true).
The significance level should be set before the hypothesis test is performed!
Common values of \(\alpha\) are 0.05 or 0.01.
| H0 is true | H0 is false | |
| Accept H0 | Type II error, miss | |
| Reject H0 | Type I error, false alarm |
Another property of a statistical test is the statistical power
power = P(Reject H0|H0 is false).
A sampling distribution is the distribution of a sample statistic. The sampling distribution can be obtained by drawing a large number of samples from a specific population.
The null distribution is a sampling distribution when the null hypothesis is true.
A null distribution
The p-value is the probability of the observed value, or something more extreme, if the null hypothesis is true.
The p-value is the probability of the observed value, or something more extreme, if the null hypothesis is true.
If the p-value is above the significance level, \(H_0\) is accepted.
If the p-value is below the significance level, \(H_0\) is rejected.
Do high fat diet lead to increased body weight?
Study setup:
The observed values, mouse weights in grams, are summarized below;
| high-fat | 25 | 30 | 23 | 18 | 31 | 24 | 39 | 26 | 36 | 29 | 23 | 32 |
| ordinary | 27 | 25 | 22 | 23 | 25 | 37 | 24 | 26 | 21 | 26 | 30 | 24 |
1. Null and alternative hypotheses
\[ \begin{aligned} H_0: \mu_2 = \mu_1 \iff \mu_2 - \mu_1 = 0\\ H_1: \mu_2>\mu_1 \iff \mu_2-\mu_1 > 0 \end{aligned} \]
where \(\mu_2\) is the (unknown) mean body weight of the high-fat mouse population and \(\mu_1\) is the mean body-weight of the control mouse population.
Studied population: Female mice that can be ordered from a lab.
2. Select appropriate significance level \(\alpha\)
\[\alpha = 0.05\]
3. Test statistic
Of interest; the mean difference between high-fat and control mice
\[D = \bar X_2 - \bar X_1\]
Mean weight of 12 (randomly selected) mice on ordinary diet, \(\bar X_1\). \(E[\bar X_1] = E[X_1] = \mu_1\)
Mean weight of 12 (randomly selected) mice on high-fat diet, \(\bar X_2\). \(E[\bar X_2] = E[X_2] = \mu_2\)
Observed values;
\(\bar x_1 = 25.83\), mean weight of control mice (ordinary diet)
\(\bar x_2 = 28.00\), mean weight of mice on high-fat diet
\(d_{obs} = \bar x_2 - \bar x_1 = 2.1667\), difference in mean weights
4. Null distribution
If high-fat diet has no effect, i.e. if \(H_0\) was true, the result would be as if all mice were given the same diet.
The 24 mice were initially from the same population, depending on how the mice are randomly assigned to high-fat and normal group, the mean weights would differ, even if the two groups were treated the same.
Random reassignment to two groups can be accomplished using permutation.
Assume \(H_0\) is true, i.e. assume all mice are equivalent and
If we repeat 1-2 many times we get the sampling distribution when \(H_0\) is true, the so called null distribution, of difference in mean weights.
4. Null distribution
5. Compute p-value
What is the probability to get an at least as extreme mean difference as our observed value, \(d_{obs}\), if \(H_0\) was true?
\(P(\bar X_2 - \bar X_2 \geq d_{obs} | H_0) =\) 0.17
6. Conclusion?
| Accept H0 | Reject H0 | |
| H0 is true | Type I error, false alarm | |
| H0 is false | Type II error, miss |
| Accept H0 | Reject H0 | |
| H0 is true | Type I error, false alarm | |
| H0 is false | Type II error, miss |
| Accept H0 | Reject H0 | |
| H0 is true | TN | FP |
| H0 is false | FN | TP |
| Accept H0 | Reject H0 | |
| H0 is true | Type I error, false alarm | |
| H0 is false | Type II error, miss |
| Accept H0 | Reject H0 | |
| H0 is true | TN | FP |
| H0 is false | FN | TP |
Significance level
\[P(\mbox{reject }\,H_0 | H_0 \,\mbox{is true}) = P(\mbox{type I error}) = \alpha\]
Statistical power
\[P(\mbox{reject } H_0 | H_1 \mbox{ is true}) = P(\mbox{reject } H_0 | H_0 \mbox{ is false}) = 1 - P(\mbox{type II error})\]
To achieve a family-wise error rate of \(\leq \alpha\) when performing \(m\) tests, declare significance and reject the null hypothesis for any test with \(p \leq \alpha/m\).
Objections: too conservative
| H0 is true | H0 is false | |
| Accept H0 | TN | FN |
| Reject H0 | FP | TP |
The false discovery rate is the proportion of false positives among ‘hits’, i.e. \(\frac{FP}{TP+FP}\).
Benjamini-Hochberg’s method control the FDR level, \(\gamma\), when performing \(m\) independent tests, as follows:
Sometimes an adjusted significance threshold is not reported, but instead ‘adjusted’ p-values are reported.
Using Bonferroni’s method the ‘adjusted’ p-values are:
\(\tilde p_i = \min(m p_i, 1)\).
A feature’s adjusted p-value represents the smallest FWER at which the null hypothesis will be rejected, i.e. the feature will be deemed significant.
Benjamini-Hochberg’s ‘adjusted’ p-values are called \(q\)-values:
\(q_i = \min(\frac{m}{i} p_i, 1)\)
A feature’s \(q\)-value can be interpreted as the lowest FDR at which the corresponding null hypothesis will be rejected, i.e. the feature will be deemed significant.
| p-value | adj p (Bonferroni) | q-value (B-H) |
|---|---|---|
| 1.7e-08 | 0.0002 | 0.0002 |
| 5.8e-08 | 0.0006 | 0.0003 |
| 3.4e-07 | 0.0034 | 0.0011 |
| 9.1e-07 | 0.0091 | 0.0020 |
| 1e-06 | 0.0100 | 0.0020 |
| 2.4e-06 | 0.0240 | 0.0040 |
| 2.3e-05 | 0.2300 | 0.0329 |
| 3.6e-05 | 0.3600 | 0.0450 |
| 0.00022 | 1.0000 | 0.2300 |
| 0.00023 | 1.0000 | 0.2300 |
| 0.00073 | 1.0000 | 0.6636 |
| 0.0032 | 1.0000 | 1.0000 |
| 0.0045 | 1.0000 | 1.0000 |
| 0.0087 | 1.0000 | 1.0000 |
| 0.0089 | 1.0000 | 1.0000 |
| 0.012 | 1.0000 | 1.0000 |
| 0.014 | 1.0000 | 1.0000 |
| 0.045 | 1.0000 | 1.0000 |
| 0.08 | 1.0000 | 1.0000 |
| 0.23 | 1.0000 | 1.0000 |